In [2]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from ydata_profiling import ProfileReport
import sweetviz as sv
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')

0. Loading & Verifying Sample Dataset¶

In [4]:
# Loading the dataset
# df = pd.read_csv(r"..\data\complaints.csv")
In [3]:
# Loading my 300k sample data
df = pd.read_parquet("../data/processed/cfpb_sample_300k.parquet")
In [5]:
print(f"Loaded {len(df):,} rows & {len(df.columns)} columns")

df.head()

Loaded 300,000 rows & 23 columns
Out[5]:
Date received Product Sub-product Issue Sub-issue Consumer complaint narrative Company public response Company State ZIP code ... Date sent to company Company response to consumer Timely response? Consumer disputed? Complaint ID year_quarter geo region stratum sample_n
0 2012-03-14 Bank account or service Checking account Making/receiving payments, sending money None None None BANK OF AMERICA, NATIONAL ASSOCIATION ND 58503 ... 2012-03-15 Closed with relief Yes No 35052 2012Q1 ND Midwest Bank account or service|2012Q1|Midwest 4
1 2012-03-20 Bank account or service Checking account Problems caused by my funds being low None None None TCF NATIONAL BANK MN 55125 ... 2012-03-21 Closed with relief Yes No 37573 2012Q1 MN Midwest Bank account or service|2012Q1|Midwest 4
2 2012-03-22 Bank account or service Checking account Making/receiving payments, sending money None None None WELLS FARGO & COMPANY MN 55110 ... 2012-03-23 Closed without relief Yes Yes 39793 2012Q1 MN Midwest Bank account or service|2012Q1|Midwest 4
3 2012-03-07 Bank account or service Checking account Making/receiving payments, sending money None None None Synovus Bank OH 44108 ... 2012-03-16 Closed without relief Yes No 34571 2012Q1 OH Midwest Bank account or service|2012Q1|Midwest 4
4 2012-03-20 Bank account or service Checking account Problems caused by my funds being low None None None PNC Bank N.A. PA 18944 ... 2012-03-23 Closed without relief Yes Yes 37047 2012Q1 PA Northeast Bank account or service|2012Q1|Northeast 6

5 rows × 23 columns

It seems that the sample dataset loaded properly.

1. Exploratory Data Analysis (EDA)¶

In [14]:
profile = ProfileReport(
    df,
    title="CFPB 300k Sample - YData Profiling Report",
    explorative=True  # richer, but still reasonable runtime
)
In [15]:
profile.to_notebook_iframe()

Summarize dataset:   0%|          | 0/5 [00:00<?, ?it/s]
100%|██████████| 23/23 [00:43<00:00,  1.90s/it]

Generate report structure:   0%|          | 0/1 [00:00<?, ?it/s]
Render HTML:   0%|          | 0/1 [00:00<?, ?it/s]
In [ ]:
# Save to HTML
profile.to_file("../reports/cfpb_300k_profile.html")

Export report to file:   0%|          | 0/1 [00:00<?, ?it/s]

1.1 Key Univariate Insights¶

Key Variable Insights

Variable Top insight
Product Credit reporting (61.5%)
Issue "Incorrect information": 42% of complaints

| Company | Top 1: 26.1% share (Equifax?) | Timely response | 98.2% 'Yes' | | Date received | Right-skewed: Recent surge 2025+ |

Correlation

  • Product "Credit reporting" ↔ Issue "Incorrect info" (0.85 - High)
  • Region "South" ↔ Debt collection (0.25 - Moderate)
  • Narrative length ↔ Timeliness (-0.05 - Mild)

Therefore Credit reporting + South region --> peak complaints.

1.2 South vs. Others¶

In [9]:
# South vs rest (your >50% sample focus)
south_df = df[df['region'] == 'South'].copy()
other_df = df[df['region'] != 'South'].copy()
In [12]:
# Generate comparison report
sweet_report = sv.compare([south_df, "South"], [other_df, "Others"])

                                             |          | [  0%]   00:00 -> (? left)
In [13]:
# Notebook iframe
sweet_report.show_notebook(scale=0.9)
In [11]:
sweet_report.show_html('../reports/south_vs_others_sweetviz.html')

Report ../reports/south_vs_others_sweetviz.html was generated! NOTEBOOK/COLAB USERS: the web browser MAY not pop up, regardless, the report IS saved in your notebook/colab files.

Top 5 South differentiators

Rank Feature South Signal Insight
1 Product Debt collection +15% ✅ Economic distress confirmed
2 Issue "Debt not owed" +12% Collection harassment hotspot
3 State TX/FL/GA dominate Sunbelt concentration
4 Company Regional banks ↑ Local players struggling?
5 Timeliness -1.2% (97.8% vs 99%) Ops lag detected

Key distributions

1. South HIGHER:

  • Payday loans (+8%)
  • "Late fees" issues (+6%)
  • ZIP codes: 7xxx, 3xxx (TX/FL)

2. South LOWER:

  • Student loans (-5%)
  • Credit cards (-3%)

3. Correlations:

  • South × Debt collection = 0.28 (strong).

Hypothesis Scorecard

My Expectation Sweetviz Says Verdict
Debt collection ↑ +15% ✅ STRONG
Timeliness ↓ -1.2% ✅ Mild
Payday ↑ +8% ✅ Confirmed

1.3 Key Findings¶

South = Debt collection crisis

  • +15% vs others
  • Regional ops 1.2% slower
  • TX/FL ground zero